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DETAILED ACTION 

1 . Claims 33-47 and 49-61 are pending. 

2. The rejection to c Claims 33-47 and 49-61 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Parker et al. (US 7024280) is withdrawn in view of new applied prior art. 

Claim Rejections - 35 USC § 102 

3. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public use or on 
sale in this country, more than one year prior to the date of application for patent m the United Stales. 

4. Claims 33-47 and 49-61 are rejected under 35 U.S.C. 102(b) as being anticipated by Wei 
et al. {Multisensory Visual servoing by a Neural Network, 1999). 

As per claim 33, Wei et al. , teaches a multisensory visual servoing by a neural network, 
including a capturing a nvimber of images of a calibration object (see fig. 4) by the camera; 
determining a set of intrinsic parameters (see page 1, col. 1, second paragraph, wherein both 
intrinsic and extrinsic parameters have been covered) of the camera (see figs. 2-3) fi-om at least 
one of the number of images of the calibration object (see fig. 4) captured by the camera; and 
determining a set of extrinsic parameters (see page 1, col. 1, second paragraph, wherein both 
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intrinsic and extrinsic parameters have been covered) of the camera (see figs. 2-3) from at least 
one of the number of images of the calibration object (see fig. 4) captured by the camera, the set 
of extrinsic parameters (see page 1, col. 1, second paragraph, wherein both intrinsic and extrinsic 
parameters have been covered) comprising a camera (see figs. 2-3) space-to-training space 
transformation defining a transformation between a camera (see figs. 2-3) space reference flame 
and a training space reference frame (see page 2, col. 1, wherein reference frame has been 
considered as position frame) . 

As per claim 34, Wei et al. . teaches a multisensory visual servoing by a neural network, 
including a positioning the camera (see figs. 2-3) with respect to the calibration object (see fig. 
4). 

As per claim 35, Wei et al. . teaches a multisensory visual servoing by a neural network, 
including a method wherein positioning the camera (see figs. 2-3) with respect to the calibration 

object (see fig. 4) comprises positioning the camera (see figs. 2-3) orthogonally with respect to a 
ruled template with a number of features (see page 5, table II, which has been considered as 
template), where a known or determinable physical relationship exists between at least some of 
the features (see page 2, col. 1). 

As per claim 36, Wei et al. , teaches a multisensory visual servoing by a neural network, 
including a method wherein positioning the camera (see figs. 2-3) with respect to the calibration 
object (see fig. 4) comprises positioning the camera (see figs. 2-3) with respect to a sample of a 
type of object (see fig. 4) the robot (see fig. 3) will manipulate, the sample having a number of 
features, where a known or determinable physical relationship exists between at least some of the 
features (see page 2, col. 1). 
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As per claim 37, Wei et al. . teaches a multisensory visual servoing by a neural network, 
including a method wherein capturing a number of images of a calibration object (see fig. 4) by 
the camera (see figs. 2-3) comprises capturing at least one image at each of a plurality of 
positions spaced perpendicularly fi-om the calibration object (see fig. 4). 

As per claim 38, Wei et al. , teaches a multisensory visual servoing by a neural network, 
including a method wherein capturing a number of images of a calibration object (see fig. 4) by 
the camera (see figs. 2-3) comprises capturing at least one image at each of a plurality of 
different orientations with respect to the calibration object (see fig. 4). 

As per claim 39, Wei et al. , teaches a multisensory visual servoing by a neural network, 
including a method wherein determining a set of intrinsic parameters (see page 1, col. 1, second 
paragraph, wherein both intrinsic and extrinsic parameters have been covered) of the camera (see 
figs. 2-3) from the at least one of the number of images of the calibration object (see fig. 4) 
captured by the camera (see figs. 2-3) comprises determining at least one of a focal length, a first 
order radial lens distortion coefficient, a set of coordinates of a center of a radial lens distortion, 
or a scale factor indicative of a framegrabber scanline resampling uncertainty. 

As per claim 40, Wei et al. . teaches a multisensory visual servoing by a neural network, 
including a method wherein determining a set of extrinsic parameters (see page 1, col. 1, second 
paragraph, wherein both intrinsic and extrinsic parameters have been covered) of the camera (see 
figs. 2-3) from at least one of the number of images of the calibration object (see fig. 4) captured 
by the camera, the set of extrinsic parameters (see page 1, col. 1, second paragraph, wherein both 
intrinsic and extrinsic parameters have been covered) comprising a camera (see figs. 2-3) space- 
to- training space fransformation defining a fransformation between a camera (see figs. 2-3) 
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space reference frame (see page 2, col. 1, wherein reference frame has been considered as 
position frame) and a training space reference frame (see page 2, col. 1, wherein reference frame 
has been considered as position frame) comprises determining a respective franslation 
component along three orthogonal axes, and a respective rotation component about the three 
orthogonal axes. 

As per claim 41, Wei et ah . teaches a multisensory visual servoing by a neural network, 
including a method that fiirther comprising: determining a camera (see figs. 2-3) space-to-tool 
space fransformation based at least in part on at least two of the number of images captured by 
the camera (see figs. 2-3) of the calibration object (see fig. 4). 

As per claim 42, Wei et al. . teaches a multisensory visual servoing by a neural network, 
including a method that fiirther comprising: determining a camera (see figs. 2-3) space-to-tool 
space fransformation based on single one of the number of images captured by the camera (see 
figs. 2-3) of the calibration object (see fig. 4) and on a number of physical coordinates of at least 
one feature of the calibration object (see fig. 4). 

As per claim 43, Wei et al. . teaches a multisensory visual servoing by a neural network, 
including a method that fiirther comprising: capturing an image of a teaching object (see fig. 4) 
of a type of object (see fig. 4) that will be manipulated by the robot; selecting a number of 
features (see page 2, col. 1) from the captured image of the teaching object (see fig. 4); 
determining a set of object (see fig. 4) space coordinates for each of the selected features (see 
page 2, col. 1) I from the captured image of the teaching object (see fig. 4). 

As per claim 44, Wei et al. . teaches a multisensory visual servoing by a neural network, 
including a method wherein selecting a number of features (see page 2, col. 1) from the captured 
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image of the teaciiing object (see fig. 4) comprises selecting six features (see page 2, col. 1) from 
the captured image of the teaching object (see fig. 4). 

As per claim 45, Wei et al. . teaches a multisensory visual servoing by a neural network, 
including a method that further comprising: 

determining an object (see fig. 4) space-to-camera (see figs. 2-3) space transformation defining a 
transformation between an object (see fig. 4) space reference frame (see page 2, col. 1, wherein 
reference frame has been considered as position frame) and the camera (see figs. 2-3) space 
reference frame (see page 2, col. 1, wherein reference frame has been considered as position 
frame). 

As per claim 46, Wei et al. . teaches a multisensory visual servoing by a neural network, 
including a method that comprising: determining a position and an orientation of an object (see 
fig. 4) frame in the tool frame reference frame (see page 2, col. 1, wherein reference frame has 
been considered as position frame) based at least in part on the object (see fig. 4) frame-to- 
camera (see figs. 2-3) space and camera (see figs. 2-3) space-to- tool space transformations (see 
absfract, pages 1-2 and fig. 2). 

As per claim 47, Wei et al. . teaches a multisensory visual servoing by a neural network, 
including a method that further comprising: providing the position and orientation of the object 
(see fig. 4) frame to the robot; and fraining an intended operation path inside the object frame 
(see abstract and fig. 4). 

As per claim 49, Wei et al. . teaches a multisensory visual servoing by a neural network, 
including a method that further comprising: adjusting a position of the movable portion of the 
robot (see fig. 3) if the number of features (see page 2, col. 1) located in the captured image of 
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the target object (see fig. 4) is determined to be an insufficient number of features; and 
capturing another two-dimensional image of the volume containing the target 1 object (see fig. 2- 
4, wherein image taken by the camera being considered has having the capacity to do two- or 
three-dimensional images) before determining the object (see fig. 4) space-to-camera (see figs. 2- 
3) space transformation for the target object (see fig. 4). 

As per claim 50, Wei et al. . teaches a multisensory visual servoing by a neural network, 
including a method that usefiil in three-dimensional pose estimation for use with a single camera 
(see figs. 2-3) mounted to a movable portion of a robot, the method comprising: 
capturing a two-dimensional image of a volume containing a target object (see fig. 4); locating a 
number of features (see page 2, col. 1) in the captured image of the target object (see fig. 4); and 
determining an object (see fig. 4) space-to-camera (see figs. 2-3) space transformation for the 
target object (see fig. 4) based at least in part on a position of at least some of the located features 
(see page 2, col. 1) using only the captured image and an algorithm that employs a known or 
determinable physical relationship between at least some of the located features (see page 2, col. 
1). 

As per claim 5 1 , Wei et al. . teaches a multisensory visual servoing by a neural network, 

including a method that further comprising: determining at least one movement of the robot (see 
fig. 3) that orients the camera (see figs. 2-3) orthogonally with respect to the target object (see 
fig. 4) based at least on part on the object (see fig. 4) space-to-camera (see figs. 2-3) space 
transformation (see abstract, pages 1-2 and fig. 2). 

As per claim 52, Wei et al. . teaches a multisensory visual servoing by a neural network, 
including a method that comprising: determining a position of the object (see fig. 4) frame in the 
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tool space reference frame (see page 2, col. 1, wherein reference frame has been considered as 
position frame); and providing an object (see fig. 4) frame to the robot (see fig. 3). 

As per claim 53, Wei et al. . teaches a multisensory visual servoing by a neural network, 
including a system that usefiil in robotics, the apparatus comprising: a single camera (see figs. 2- 
3) operable to capture at a number of images of a calibration object (see fig. 4) means for 
calibrating the camera, by: determining a set of intrinsic parameters (see page 1, col. 1, second 
paragraph, wherein both intrinsic and extrinsic parameters have been covered) of the camera (see 
figs. 2-3) from at least one of the number of images of the calibration object (see fig. 4) captured 
by the camera; and determining a set of extrinsic parameters (see page 1, col. 1 , second 
paragraph, wherein both intrinsic and extrinsic parameters have been covered) of the camera (see 
figs. 2-3) from at least one of the number of images of the calibration object (see fig. 4) captured 
by the camera, the set of extrinsic parameters (see page 1, col. 1, second paragraph, wherein both 
intrinsic and extrinsic parameters have been covered) comprising a camera (see figs. 2-3) space- 
to-fraining space transformation defining a fransformation between a camera (see figs. 2-3) space 
reference frame (see page 2, col. 1, wherein reference frame has been considered as position 
frame) and a fraining space reference frame (see page 2, col. 1, wherein reference frame has been 
considered as position frame); and means for estimating a pose of a target object (see fig. 4), by: 
capturing a two-dimensional image of a volume containing a target object (see fig. 4); and 
locating at least six features (see page 2, col. 1) in the captured image of the target object (see 
fig. 4); and determining an object (see fig. 4) space-to-camera (see figs. 2-3) space 
fransformation based at least in part on a position of at least some of the located features (see 
page 2, col. 1) in solely the captured image using an algorithm that employs a known or 
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determinable physical relationship between at least some of the located features (see page 2, col. 
1). 

As per claim 54, Wei et al. . teaches a multisensory visual servoing by a neural network, 
including a system that comprising: means for training, comprising: 

capturing an image of a teaching object (see fig. 4) of a type of object (see fig. 4) that will be 
manipulated by the robot; selecting a number of features (see page 2, col. 1) fi-om the captured 
image of the teaching object (see fig. 4); determining a set of object (see fig. 4) space coordinates 
for each of the selected features (see page 2, col. 1) fi-om the captured image of the teaching 
object (see fig. 4); an determining an object (see fig. 4) space-to-camera (see figs. 2-3) space 
transformation defining a transformation between an object (see fig. 4) space reference fi-ame 
(see page 2, col. 1, wherein reference fi-ame has been considered as position fi-ame) and the 
camera (see figs. 2-3) space reference fi-ame (see page 2, col. 1, wherein reference frame has 
been considered as position frame). 

As per claim 55, Wei et al. . teaches a multisensory visual servoing by a neural network, 
including a system wherein the means for calibrating, the means for estimating a pose, and the 
means for fraining comprises at least one programmed computer (see absfract, page 1, col. 1 and 
page 4, col. 2, second paragraph). 

As per claim 56, Wei et al. , teaches a multisensory visual servoing by a neural network, 
including a system wherein the means for calibrating, the means for estimating a pose, and the 
means for fraining comprises at least one computer-readable medium (see absfract, page 1, col. 1 
and page 4, col. 2, second paragraph) storing instructions operating at least one computer (see 
absfract, page 1, col. 1 and page 4, col. 2, second paragraph). 
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As per claim 57, Wei et al. , teaches a multisensory visual servoing by a neural network, 
including a system wherein the pose estimating means estimates the pose of the target object (see 
fig. 4) further by: adjusting a position of the movable portion of the robot (see fig. 3) if the 
number of features (see page 2, col. 1) located in the captured image of the target object (see fig. 
4) is determined to be an insufficient number of features (see page 2, col. 1). 

As per claim 58, Wei et al. . teaches a multisensory visual servoing by a neural network, 
including a system that useful in robotics, the apparatus comprising: 

a single camera (see figs. 2-3) operable to capture a number of images of a calibration object (see 
fig. 4) means for calibrating the camera, by: determining a set of intrinsic parameters (see page 1, 
col. 1, second paragraph, wherein both intrinsic and extrinsic parameters have been covered) of 
the camera (see figs. 2-3) from at least one of the number of images of the calibration object (see 
fig. 4) captured by the camera; and determining a set of extrinsic parameters (see page 1, col. 1, 
second paragraph, wherein both intrinsic and extrinsic parameters have been covered) of the 
camera (see figs. 2-3) from at least one of the number of images of the calibration object (see fig. 
4) captured by the camera, the set of extrinsic parameters (see page 1, col. 1, second paragraph, 
wherein both intrinsic and extrinsic parameters have been covered) comprising a camera (see 
figs. 2-3) space-to-training space transformation defining a transformation between a camera (see 
figs. 2-3) space reference frame (see page 2, col. 1, wherein reference frame has been considered 
as position frame) and a training space reference frame (see page 2, col. 1, wherein reference 
frame has been considered as position frame) ; and means for estimating a pose of a target object 
(see fig. 4), by: capturing a two-dimensional image of a volume containing a target object (see 
fig. 4); locating at least five features (see page 2, col. 1) in the captured image of the target object 
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(see fig. 4); and determining an object (see fig. 4) space-to-camera (see figs. 2-3) space 
transformation based at least in part on a position of at least some of the located features (see 
page 2, col. 1) using the captured image without any additional captured images and an 
algorithm that employs a known or determinable physical relationship between at least some of 
the located features (see page 2, col. 1). 

As per claim 59, Wei et al. . teaches a multisensory visual servoing by a neural network, 
including a system wherein the means for calibrating and the means for estimating a pose 
comprises at least one programmed computer (see abstract, page 1, col. 1 and page 4, col. 2, 
second paragraph). 

As per claim 60, Wei et al. . teaches a multisensory visual servoing by a neural network, 
including a system wherein the means for calibrating and the means for estimating a pose 
comprises at least one computer-readable medium (see abstract, page 1, col. 1 and page 4, col. 2, 

second paragraph) storing instructions operating at least one computer (see abstract, page 1, col. 
1, wherein "traditional computer" inherently works with computer-readable medium, and page 4, 
col. 2, second paragraph). 

As per claim 53, Wei et al. . teaches a multisensory visual servoing by a neural network, 
including a system wherein the pose estimating means estimates the pose of the target object (see 
fig. 4) further by: adjusting a position of the movable portion of the robot (see fig. 3) if the 
number of features (see page 2, col. 1) located in the captured image of the target object (see fig. 
4) is determined to be an insufficient number of features (see page 2, col. 1). 
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Response to Arguments 

5. As to the reference dos not use video images in autonomous mode (see figs. 2-3); 
As to the sensor 1 14 not emit a signal (see fig. 2); 

As to the reference not teaching a calibration of the video device 122 "or any other part 
of the robot" (see abstract); 

As to the reference not teaching a robot (see fig. 3) that can grip object (see figs. 2-4)s in 
autonomous mode (see fig. 3); 

6. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to MCDIEUNEL MARC whose telephone number is (571)272- 
6964. The examiner can normally be reached on 6:30-5:00 Mon-Thu. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Khoi Tran can be reached on (571) 272-6919. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 
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/McDieunel Marc/ 
Examiner, Art Unit 3664 
August 24, 2009 

/KHOI IRAN/ 

Supervisory Patent Examiner, Art Unit 3664 



